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ABSTRACT 

This paper presents the paradigm of interactive and co- 
operative sensing and control as a fundamental mecha- 
nism of integrating and fusing the strengths of man and 
machine for advanced teleoperation. The interactive and 
cooperative sensing and control is considered as an ex- 
tended and generalized form of traded and shared con- 
trol. The emphasis of interactive and cooperative sensing 
and control is given to the distribution of mutually non- 
exclusive subtasks to man and machine, the interactive 
invocation of subtasks under the man/machine symbiotic 
relationship, and the fusion of information and decision- 
making between man and machine according to their 
confidence measures. The proposed interactive and co- 
operative sensing and control system is composed of 
such major functional blocks as the logical sensor sys- 
tem, the sensor-based local autonomy, the virtual envi- 
ronment formation, and the cooperative decision-making 
between man and machine. The Sensing-Knowledge- 
Command (SKC) fusion network is proposed as a funda- 
mental architecture for implementing cooperative and in- 
teractive sensing and control. Simulation results are 
shown. 

INTRODUCTION 

Early attempts on teleoperation were based on tight cou- 
pling between the manipulator and the operator through 
mechanical linkages or steel tapes, as is the case of the 
AEC Argonne Laboratory series *, or electrical or hy- 
draulic connections, as is the case of the GE telemanipu- 
lators built by Mosher 2 . 

The telemanipulation based on the direct coupling be- 
tween man and machine severely limits its performance 
: it neither accommodates the desirable mechanical dex- 
terity due to the difficulty of manually coordinating mul- 
tiple joints, nor allows high task complexity due to the 
difficulty of achieving required compliance. It gives an 


excessive burden on the operator, which may cause long 
task completion time with a high failure rate. 

The need to improve mechanical dexterity in teleopera- 
tion and achieve desirable compliance during teleopera- 
tion, so as to deal with more complex tasks under a 
partially constraint environment but with the comfort of 
h uman operator, has prompted the development of the 
following teleoperation paradigms : 

1) The generalized bilateral telemanipulation 3 ’ 4 ’ 

5> 6 in which the tight and one-directional cou- 
pling between the master and the slave is re- 
placed by loose and two-directional coupling 
characterized by computer-based bilateral infor- 
mation transformation and exchange. This al- 
lows that the slave arm may not need to be the 
exact kinema tic replica of the master arm, and 
that the operator can feel the contact force felt by 
the slave arm through force feedbacks, which al- 
lows human to execute compliance control. 

2) The supervisory control with shared and trad- 
ed control 7 > 8 , in which a task is decomposed 
into temporarily (traded control) or spatially 
(shared control) disjoint subtasks that are to be 
distributed to man and machine. For instance, the 
operator can be supported by software jigs or 
spatial support means 9, 10 which take advantages 
of spatial constraints in the task to allow the 
slave manipulator to control those degrees of 
freedom specified by the motion constraint, 
while the operator controls the rest of degrees of 
freedom. Or, the slave arm with force/torque sen- 
sors is responsible for automatic compliance 
control, while the operator is responsible for the 
motion control. The supervision of telemanipu- 
lation 11 is done by the supervisory loop closed 
through the human operator, for which visual 
and graphic displays and force reflections from 
the remote site play an important role. 
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The recent advancement in the theory and practice of ro- 
botics and intelligent systems makes it necessary to ex- 
ploit new generation of teleoperation which fully utilizes 
the high degree of mechanical dexterity provided by re- 
dundant and multiple arms and the capability of a robot 
performing sensor-based local autonomy. Especially, the 
role of man and machine should be redefined for ad- 
vanced teleoperation in such a way that the slave arm be- 
comes an active partner of the human operator, 
supporting perception, decision-making, and coopera- 
tive task execution. Tb achieve this requires to explore a 
fundamental mechanism of integrating and fusing the 
strengths of man and machine for advanced teleopera- 
tion. 

This paper presents a paradigm of interactive and coop- 
erative sensing and control as the fundamental mecha- 
nism of integrating and fusing the strengths of man and 
machine for advanced teleoperation. The interactive and 
cooperative sensing and control is considered as an ex- 
tended and generalized form of traded and shred control. 
The emphasis of interactive and cooperative sensing and 
control is given to the distribution of mutually nonexclu- 
sive subtasks to man and machine, the interactive invo- 
cation of sub tasks to achieve the man/machine symbiotic 
relationship, and the fusion of information and decision- 
making between man and machine according to their 
confidence measures. 

THEORY OF INTERACTIVE AND CO- 
OPERATIVE SENSING AND CONTROL 

The quality of teleoperation depends on the performance 
of the operator in perceiving and understanding task 
mechanisms correctly and in generating control com- 
mands precisely in consistency with his/her perception 
and intention. The quality of teleoperation also depends 
on the performance of the machine (as a master-slave 
system) in providing accurate and sensitive control 
which is stable and robust under disturbances, system 
nonlinearities, and time delays. 

As a means of enhancing the performance of the opera- 
tor, there have been developed methods for accomplish- 
ing powerful telepresence based on sensory feedbacks 
using visual displays and force reflections, as well as 
methods for effectively training the operator to achieve 
the high level of expertise. On the other hand, the devel- 
opment of advanced teleoperator controllers based on die 
concept of impedance, passivity, dynamic coordination, 
and predictive modeling has been pursued as a means of 
improving the performance of the machine. 

However, there exist fundamental limitations for the op- 


erator to achieve accurate perception of task geometries 
and control behaviors and, even more so, to accomplish 
precise coordination between perception and action. This 
is mainly due to fire impreciseness and low bandwidth in 
human sensory-motor coordination: human depends 
heavily on sensor-based adaptive motion corrections to 
compensate for imprecise positioning and is unable to re- 
spond to high bandwidth tasks. And, partly due to the dif- 
ficulty of implementing powerful telepresence as well as 
high performance of control. 

The best way of relaxing the above limitations is to fully 
utilize the strengths of man and machine in such a way as 
to achieve the mutual compensation of individual weak- 
nesses. The strength of human lies in understanding task 
mechanisms, recognizing objects, generating task and 
motion plans under global constraints, whereas the 
strength of machine lies in precision positioning, quanti- 
zation of primitive features, repetition of memorized 
tasks, and sensor-based local reflex. Attempts have been 
made to incorporate the strengths of man and machine in 
teleoperation: traded control temporally decompose a 
task and assigned to human and machine according to 
whether human or machine fits for a give subtask, while 
shared control spatially decompose a task into subtasks 
to be carried out by man and machine simultaneously. An 
instance of shared control is that compliance control is 
automatically accomplished by machine based on sensed 
forces, while position control is done through operator’s 
manual control. 

Although traded control and shared control provide a 
means of combining the strengths of man and machine, 
they do not present a general and powerful methodology 
of integrating man and machine. This is because traded 
and shared control is based on clear-cut decomposition 
of tasks into subtasks to be distributed individually to 
man and machine, where such decomposition is often 
difficult to achieve, resulting in overly simplified distri- 
bution of a task. More importantly, such a clear-cut de- 
composition eliminates the possibility of fusing multiple 
sources of information and decision-making from man 
and machine. 

We propose interactive and cooperative sensing and con- 
trol as a fundamental paradigm of integrating and fusing 
the strengths of man and machine for teleoperation. The 
interactive and cooperative sensing and control is an ex- 
tended and generalized form of traded and shared con- 
trol. The emphasis of interactive and cooperative sensing 
and control is given to the distribution of mutually non- 
exclusive subtasks to man and machine, the interactive 
invocation of subtasks with symbiotic relationship, and 
the fusion of information and decision-making from man 
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Fig.l Interactive and Cooperative Sensing and Control for Advanced Teleoperation. 


and machine according to the degree of their confidence. 

The interactive and cooperative sensing and control con- 
sists of the following major functional blocks : 1) logical 
sensor system. 2) sensor-based local autonomy.3) virtual 
environment formation. 4) cooperative decision-making 
between man and machine. 

Logical Sensor System 

A logical sensor represents, in an abstract form, one of 
the many functional capabilities that the integrated sen- 
sor system can provide. The distance sensor, the surface 
orientation sensor, the forceAorque sensor, the feature 
finding sensor, eta are a few examples of logical sensors. 
There may or may not exist a direct association between 
a logical sensor and a physical sensor, such that a logical 
sensor can achieve its goal (to generate its output) based 
on the outputs of other logical sensors and/or physical 
sensors. Logical sensors can be hierarchically organized 
into a logical sensor system based on their functional in- 
terdependency. A logical sensor system not only pro- 
vides a symbolic list of the various perceptual 
capabilities of a robot, but also represents a number of 
different ways of accomplishing the goal of a logical sen- 
sor. The latter is especially useful for sensor fusion. The 
symbolic representation of a logical sensor system pro- 


vides an effective tool for the intelligent interface with 
the operator performing interactive and cooperative 
sensing. For instance, a logical sensor can be invoked by 
the operator in response to the system’s request for pro- 
viding sufficient information for a sensor-based automat- 
ic operation or a virtual environment formation initiated 
by the operator. 

Sensor-Based Automatic Operations 

Sensor-based automatic operations are for providing the 
manipulator with the capability of local autonomy, such 
that man/machine cooperative control can be accom- 
plished. A list of sensor-based automatic operations are 
predefined, out of which the operator can select and in- 
voke a desired sensor-based automatic operation. Exam- 
ples of sensor-based automatic operations include 
automatic tracking, automatic centering, automatic 
a ligning , automatic compliance, etc. Once invoked, it is 
sent to the interpreter to transform it into a sequence of 
actions executable by the manipulator, during the pro- 
cess of interpretation, the interpreter automatically in- 
quires the logical sensor system and/or the operator for 
foe information necessary for the complete specification 
of the corresponding sensor-based operation. The opera- 
tor performs, if necessary, sensor planning and interac- 
tive sensing, and invokes logical sensors. 
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Virtual Environment Formation 

Virtual environment formations are for providing the op- 
erator with an artificially created environment (called 
virtual environment) which enhance the operator’s un- 
derstanding of control environment and task mechanism, 
and, consequently, improve the fidelity of operator’s 
manual control. The generated virtual environment pro- 
vides a guidance and assistance for operator’s manual 
control. A list of virtual environment operations are pre- 
defined, out of which the operator can select and invoke 
a desired virtual environment operation. Virtual environ- 
ment operations generates displays or reflects forces 
which partially or fully inform the operator of the task 
specifications obtained by logical sensors for sensor- 
based automatic operations, or provide sensory feedback 
indicating the discrepancy from the sensor-based auto- 
matic operation. Examples include the surface normal 
display, the virtual force field in free space, the display of 
desired end effector orientations, etc. As is done for sen- 
sor-based operations, once invoked, it is sent to the inter- 
preter to transform it into a detailed sequence of 
operations with interactive information collection. A vir- 
tual environment formation may or may not accompany 
with the corresponding sensor-based automatic opera- 
tioa 

Man/Machine Cooperative Decision-mak- 
ing 

Since it is allowed that sensor-based automatic opera- 
tions and operator’s manual operations carry out mutual- 
ly non-exclusive tasks, we need to provide a mechanism 
for fusing two different source of decisions, or, simply 
decision fusion. The degree that individual decisions 
contribute to the final (optimal) decision should depend 
on their credibility. The credibility of decision by the ma- 
chine can be estimated in terms of the certainties in- 
volved in the sensor measurements, the decision-making 
rules, and the constraints used in the decision making. 
Whereas, the credibility of decision by the operator de- 
pends on the level of expertise obtained by the experi- 
ence. However, it should be noted that such credibilities 
are subject to variation not only with respect to time but 
also with respect to control situations: For instance, in 
case a jamming situation occurred in the peg-hole inser- 
tion process, the operator’s capability of making an error 
correction operation based on a global planning may be 
more dependable than the solution based on the sensor- 
based automatic insertion process. To handle this varia- 
tions, the operator is allowed to set the degree of contri- 
bution of individual decisions heuristically. 


Information Flow 

Figl. illustrates the information flow between the major 
functional blocks of the interactive and cooperative sens- 
ing and control system. The information flow can be 
summarized as follows: 

1) Given a task, the operator may invoke the 
sensor-based automatic operation and/or virtual 
environment formation, by selecting a menu 
from the prespecified lists. 

2) The operator can also select the system control 
mode as manual control, shared control, cooper- 
ative control, or automatic control, by simply ad- 
justing the relative weight between the sensor- 
based automatic operation and the manual oper- 
ation in cooperative decision-making. It should 
be noted that the sensor-based automatic opera- 
tions can be used solely for the purpose of virtual 
environment formation, without participating in 
cooperative control, in case the operator invokes 
both the sensor-based automatic operation and 
the virtual environment formation, but assigns 
zero weight to the sensor-based automatic oper- 
ation in cooperative decision-making. 

3) Prior to the invocation of the sensor-based au- 
tomatic operation module or the virtual environ- 
ment formation module, the operator may need 
to perform sensor planning to ensure that the in- 
voked operation can retrieve correct information 
from the logical sensor system. The interpreter of 
the sensor-based automatic operation or the vir- 
tual environment formation generates executable 
commands by filling out the existing templates 
through the interaction with the logical sensor 
system and/or the operator. 

4) The virtual environment formation module 
provides the operator with the information repre- 
senting the current control situation, especially 
in terms of the deviation of manual control from 
the sensor-based automatic operation, based on 
the multi-media interface using graphic displays, 
Cartesian space force fields at the operator’s 
hand, and sound. The virtual environment for- 
mation offers, among other things, the visual ser- 
voing guidance and the virtual compliance 
which keep the manipulator from moving away 
from the desirable pose. 
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Fig.2 A part of the logical sensor system chosen for illustration 


Scenario 

To explain the above concept in more detail, a typical 
scenario of interactive and cooperative sensing and con- 
trol for advanced teleoperation is described in the follow- 
ing based on the peg-hole insertion task: 

- Let us assume that the manipulator in a remote 
site has various sensors such as proximity sen- 
sors, force/torque sensors, tactile sensors, and a 
mini-camera mounted on the end effector, as 
well as stereo cameras fixed in space for the pur- 
pose of globally monitoring the task space. The 
capabilities of the above sensors can be summa- 
rized and organized in a logical sensor system, 
e.g., as shown in Fig2. Each logical sensor has its 
own sensing goal to be achieved through the log- 
ical sensor hierarchy. The data that a logical sen- 
sor represents is associated with a confidence 
measure to be used in sensor fusion, which may 
occur when multiple paths of achieving the sens- 
ing goal exists in the logical sensor hierarchy, 
and in cooperative decision-making. 

- With the aid of the various sensors mounted on 
the end effector, the manipulator is able to per- 
form various simple sensor-based automatic op- 
erations: maintaining orientations, tracking 
predefined features, reaching identified posi- 
tions, reacting to contact forces for compliance, 
centering on a geometric feature, aligning to a 
surface normal, etc. These sensor-based auto- 
matic operation primitives require a minimal op- 
erator intervention for interpretation. For 
instance, the “Align Surface Normal” primitive 
requires the operator to position the end-effector 
near the corresponding surface prior to the invo- 


cation of the primitive. The executable command 
will then be automatically generated by the inter- 
preter filling out the corresponding template 
through the interaction with the logical sensor 
system, and/or the human operator. 

- Let us also assume that the system is capable of 
providing the operator with virtual environments 
based on visual displays using video images and 
graphics, 3D force field at the operator’s hand, 
and sound. The virtual environment can be 
formed by representing the discrepancy between 
the sensor-based automatic operation and the op- 
erator’s manual operation. In fact, a sensor-based 
automatic operation can be invoked solely for 
the purpose of virtual environment formation, 
should the operator desired to do so. Other list of 
virtual environment include a force field about 
surface normals, a graphic overlay of command- 
ed manipulator configuration on the video im- 
age, a graphic display of contact force and 
moment, etc. 

- Now, let us consider that the operator is given a 
peg and hole insertion task, where the hole is as- 
sumed to have very small tolerance. The major 
difficulty of the above peg-hole teleoperation lies 
in the operator’s generation of accurate peg mo- 
tion with correct peg orientation and position. 
Especially, maintaining correct peg orientations 
throughout the insertion process is considered 
vital for avoiding jamming, but often not so easy 
to be achieved by the human operator. 

- Thus, the operator can invoke “Align Surface 
Normal” for a sensor-based automatic operation 
as well as a virtual environment formation, so 
that not only the force field about the surface nor- 
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mal, generated by the virtual environment for- 
mation module, guides the orientation of the 
operator’s motion, but also control decision is 
shared by the operator and the sensor-based au- 
tomatic operation module according to their 
strengths. 

SENSING-KNOWLEDGE-COMMAND 

FUSION 

The mechanism of sensor data fusion 12 ’ 13,14,15,16 ’ 17 can 
provide a fundamental means for achieving system inte- 
gration since it combines multiple uncertain sensor data 
into more accurate and reliable estimates, identifies 
faulty sensors through consensus verification, and main- 
tains consistency with existing constraints. We extend 
the notion of "sensor data fusion" toward a more general 
concept of " Sensing-Knowledge-Command(SKC) fu- 
sion" to include the integration of feature transformation 
and abstraction, data and concept fusion, knowledge 
propagation for consistency satisfaction and cooperative 
planning and decision-making. 

The "SKC fusion network" provides a fundamental ar- 
chitecture for implementing cooperative and interactive 
sensing and control for advanced teleoperation system 18 . 
The SKC fusion network establishes the mechanism of 
achieving network consistency in real-time through dy- 
namic evolution of network states: once invoked by in- 
puts or stimuli, the SKC fusion process enforces the 


network to converge to new equilibrium states through 
the network dynamics of data fusion, feature transforma- 
tion, and constraint propagation. The cooperative control 
of man/machine systems is then accomplished through 
the SKC fusion process invoked by stimuli from both hu- 
man and machine, where sensing, knowledge, and com- 
mand of a human and a machine are tapped into the 
network to provide inputs or stimuli for the network. 

SKC Fusion Network 

"SKC fusion network" represents a fundamental robotic 
architecture based on which the real-time connection be- 
tween perception and action is accomplished. The SKC 
fusion network is formed by the interconnection of four 
basic modules: the data fusion module, the feature trans- 
formation module, the constraint module, and the action 
module, as shown in Fig. 3. A data fusion module(DFM) 
takes one or more data representing an object feature and 
produces the optimal estimate for the feature in coopera- 
tion with the initial state of the module. A feature trans- 
formation module(FTM) extracts a primitive features 
from the raw sensory data or transforms a set of p rimi tive 
features into the more abstract, higher level features. An 
action module(AM), as a special case of a feature trans- 
formation module, issues the command to the environ- 
ment based on the predefined laws triggered by a set of 
features. A constraint module(CM) represents system 
knowledge which put a constraint upon a set of feature 
values associated with the knowledge: the feature values 
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should be adjusted in such a way as to achieve a maxi- 
mum consistency with the associated knowledge. The 
output of each module indicates the current estimates of 
the corresponding feature or knowledge, and is kept as 
the current state of the module. The state transition of a 
module propagates in both directions (forward and back- 
ward), and invokes the state transition of other modules 
having functional relationship with it. In this sense, the 
interconnection among modules is considered bidirec- 
tional, as represented in Fig. 3 by a feedback loop asso- 
ciated with each module. The domain knowledge is 
embedded in the network in two ways: explicitly by the 
constraint module, and implicitly by the functions of 
feature transformation modules as well as the network 
structure. 
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Network Dynamics 

The mechanism of SKC fusion network can be interpret- 
ed in terms of two operational modes: the forward mode 
and the backward mode. The forward mode first extracts 
primitive features from sensor data through low level 
feature transformation modules, and subsequently pro- 
duces more abstract form of features through higher-lev- 
el feature transformation modules. The forward mode 
also allows the data and concept fusion to occur through 
data fusion modules, whenever multiple and redundant 
data are available for a single feature or concept The 
backward mode starts to operate upon the activation of a 
constraint module: based on the error detected at the con- 
straint module, all the feature values connected to that 
constraint module are adjusted to satisfy consistency. 
The new updated feature values (as the output of feature 
transformation modules) in turn invokes the adjustment 
of lower level features connected to the module. Through 
a cycle of forward and backward information propaga- 
tions, the network reaches an equilibrium state, i.e., all 
the features and concepts have consistent estimates 
which are optimal in the sense that redundant sources of 
information are fused under the constraints provided by 
system knowledge. 

The entities of the SKC fusion network, such as data, fea- 
tures, concepts, and knowledge, are represented by their 
nominal values or equations and the degree of uncertain- 
ties associated with the normal values or equations. 
Thus, during a cycle of forward and backward informa- 
tion propagations, not only the nominal values or equa- 
tions but also the degree of their uncertainties need to be 
adjusted. Probabilistic modeling and inference can pro- 
vide a means of achieving the adjustment of the nominal 
values or equations, and the degree of their uncertain- 
ties. For instance, in the forward process, the output of a 
FTM can be characterized by a random variable, x, 


Fig. 4 A simple SKC Fusion Network used 
for the Description of Network Dynamics 

where the probability density function, p(x), of x is deter- 
mined based on the input random variable, s, of a known 
probability density function and the corresponding fea- 
ture transformation function, x = t(s). The output, y, of a 
DFM can be determined based on the maximum likeli- 
hood estimate, the successive Bayes estimate, and the 
minimum variance estimate.The backward process for a 
CM or a FTM can be accomplished by the nonlinear op- 
timization or the inverse mapping paradigm based on in- 
put update rule. The backward process for a DFM can be 
accomplished simply by the direct propagation of the 
output to individual inputs. The problem associated with 
the above approach based on successive computation of 
forward and backward propagation is that it is not suit- 
able for real-time implementation due to the computa- 
tional complexity involved in the processes, as well as 
the difficulty of processing non-Gaussian signals gener- 
ated by non-linear transformations. Therefore, in this pa- 
per, we present a new approach for accomplishing 
forward and backward processes of individual modules 
simultaneously and concurrently, based on the dynamic 
evolution of the states of modules. The approach is based 
on representing the SKC network as a dynamic system in 
which the network dynamic state is evolving toward the 
equilibrium state, once invoked by input stimuli, as de- 
scribed in more detail in the following. 

Dynamics of SKC Fusion Network 

For the clear description of the concept of dynamic evo- 
lution of the SKC fusion network, let us consider a sim- 
ple SKC fusion network illustrated in Fig. 4. Let us 
assume that, upon the stimuli given to the network, 
FTM1 and FTM2 have new inputs s x and s 2 , and s 3 and 
s 4 , respectively. This involves the states of individual 
modules simultaneously evolve toward the new equilib- 
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rium states that maintains network consistency. We pro- 
pose that the evolution of system states is governed by 
the following dynamics: 

1) The DFM-CM dynamics: 

yi = * x yi 

Yyi V((> {z° - g(yj, y^} (1) 

2) the FTM-DFM dynamics: 

Xi = -X x i{xi- tl ( s i,s 2 )}- 

Y X 1 { Yi ~ x 2» (2) 

x 2 = - X x2{ x 2* t2 ( s 3» s 4)>- 

Yx2 V ^^yi* fl ( x l’ x 2)} 

where the initial conditions are given by die equilibrium 
states. 

(1) and (2) represent a set of fundamental dynamic equa- 
tions governing the behavior of the SKC fusion network 
in reaching a new network equilibrium state. The first 
term of a dynamic equation represents the forward pro- 
cess, whereas the second term represents the backward 
process. To deal with mote complex networks, we need 
to simply repeat the same form of dynamic equation used 
for (1) and (2) for individual module of network with the 
proper assignment of module functions and coefficients. 

The variation of y y due to the forward process and the 


variation of y x due to the backward process should be 
determined in terms of the uncertainty associated with 
the forward process and the backward process. These 
variations can be controlled by the ratio between the co- 
efficients, V Y yl > Y xl and y^. It is possible that the 
above dynamic coefficients can be assigned in such a 
way that the result of dynamic evaluation approximately 
matches the result from a probabilistic model. In fact, the 
above dynamic equations can be considered as a general 
form of the minimum variance estimate described previ- 
ously. This observation allows not only to obtain the op- 
timal dynamic coefficients but also to update the 
uncertainties (represented by covariance matrices) in- 
volved in individual states. 

SIMULATION 

To demonstrate the operation of the SKC fusion network 
based on network dynamics given by (1) and (2), we 
chose the following simple example. A robot is given a 
task to pick up a right triangle among many different 
shapes of triangles. The robot is assumed to have two 

logical sensors: one for the measurement of edge nor- 
mals (called the "edge-normal sensor") and the other for 
the measurement of internal angles (called the "angle 
sensor"). The angle sensor is easier to handle, but has 
more uncertainty than the edge-normal sensor. For a giv- 
en triangle, the robot measures each angle twice with the 
angle sensor and takes the average of two measurements. 
On the other hand, the robot measures the edge normals 
with the edge-normal sensor, and computes the internal 
angles from the measured edge normals. Then, a decision 



Fig.4 The SKC Fusion Network for the Task to Pick-Up the Right Triangle 
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is made on the size of each angle based on data from both 
sensors. If any of three angles is a right angle, the "pick- 
up" command is issued. The sum of three internal angles 
is subject to be 180 degrees. It is assumed that the sensor 
data have the known, independent Gaussian distribu- 
tions. 

For the task described above, we can organize the SKC 
fusion network as shown in Fig. 4. The two sensed data 
from the angle sensor, sa x and sbp are fused into y., 

through DFMp VTM { takes the edge normals sn. as its 
input, and computes the internal angle, between those 

two edges. Through the feature transformation equation, 
these x- and y^ are in turn fused into the angle estimates, 

z-, through the higher level DFM^*. CM checks if z- sat- 
isfy the constraint, i.e., the sum of internal angles is 180 
degrees. As mentioned in section 3, if an error is detected 
in CM, it is used to adjust Zj, and propagated backward 

to adjust Xj and y^ The variances of Xj, y v and z i can be 

computed from those of sensor data, based on the as- 
sumption of the independent, Gaussian distribution. The 
feature transformation function in FTMj, tj(.) is defined 

as: 

tj(sn) = (180 - (snj - sn^)) MOD 360) 

where s^ and sdj are two edge normals, and we simply 

use the averaging function as the data fusion function, 
q(.), i.e., 

where s is the input vector with dimension M. 

Based on (1) and (2), the dynamic equation for z i is for- 
mulated as: 

z i = - T zi z i + (VXiV 2 +O zi*( 180 -2j=l,3 Z j>’ 
for i=l,2,3. 

The second term of the right hand side comes from the 
forward process through DFMj, whereas the third term 

from the error in CM. Since the standard deviation is 
used for the coefficient of the CM output error, with a 
smaller variance of z- is less affected by the output error 
of CM. Similarly, the dynamic equations for Xj and y- are 
formulated as: 



Fig. 5 The Sample Triangle and the sensing location 

xj = -T ri xj + ((180 - (snj-snj)) MOD 360) + 
o x j*(z i - x i), for i=l,2,3 

yj = -Tyj y { + (saj+sbj) / 2 + Oyj * (zj - yj), for i=l,2,3 

Note that the error is defined here as the difference of z i 
from y ; and Xj, since the input and output of DFM should 
be equal when an equilibrium state is reached. 

Although the SKC fusion network shows the different re- 
sult according to the input, a typical result for a triangle 
in Fig. 5 is shown in Fig. 6 with the sensor statistics and 
data in Tab. 1 The initial equilibrium state is chosen as 
the ideal data for the equilateral triangle in which all the 
x, y, and z are [60.0 60.0 60.0] without any errors in CM 
and DFMsl. The sensed edge-normal data vector, sn, is 
quite accurate due to its small variance while the sensed 
angl e data vector deviates a lot from the actual data. 
Starting from the initial equilibrium state, the top level 
estimate vector of angles, x, converges to the equilibri- 
um states, Xg = [59.6 89.4 31.5], which is very close to 

the real accurate value, [60.0 90.0 30.0] as shown in a). 
The error between x and the real value decreases gradu- 
ally and finally converges to a small value as shown in b) 

The error in CM remains as almost zero all the time as 
shown in c), and the errors in DFMs grow during the 
transient state, and then converge to near zero as shown 
in c) and d). Note that x and z are almost same in the new 
equilibrium state while the deviation of y from z is big. 
This is because there exist considerable errors in the data 
from the angle sensor, but y has been adjusted toward the 
real value in the new equilibrium state. 

To explore the effect of the backward process, the above 
simulation is repeated with change that the backward 
process is invoked at time t=2, and its results are shown 
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Fig. 6 Simulation results for the triangle in Fig. 5 Fig. 7 Simulation results for the triangle in Fig. 5 


with the parameter in Tab.l. 


with Backward Process started at t=2. 
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Actual Values 

i=l 

2 

3 

. Edge Normal, r^: 

60 

150 

300 

. Angle, ctjt 

60 

90 

30 

Sensed Data 

. Edge Normal, sn^ 

59.5 

151.0 

298.5 

. Angle, sajt 

71.5 

101.5 

39.5 

sbjt 

68.5 

98.4 

41.5 

Variance of Sensor 

angle: 100, 

edge-normal:25 

Time Constant: 

It 

tl 


T : .= 10 
21 


Tab.l Actual and Sensed Data, and Sensor Parameters 


in Fig. 7. Only with the forward process, x converges to 
equilibrium value which deviates from the real value 
considerably. However, it moves to another equilibrium 
value which is very close to the real value, just after the 
beginning of the backward process, as shown in a). The 
error between x and the real value is drawn in b) which 
shows clearly the error correcting effect of the backward 
process. The graphs in c), d) and e) also show the adjust- 
ing effects of the backward process on the errors in CM 
and DFMs. 

CONCLUSION 

This paper presents a theory of interactive and coopera- 
tive sensing and control as a fundamental paradigm of 
implementing advanced teleoperation. The proposed 
paradigm was intended to take full advantage of the cur- 
rent and future capabilities of a robot performing dex- 
trous manipulation and sensor-based local autonomy. 

A new method of achieving sensing-knowledge-com- 
mand (SKC) fusion was presented as a basic computa- 
tional mechanism for the proposed interactive and 
cooperative sensing and control. 

A system architecture and man/machine interface proto- 
col was described to show the preliminary implementa- 
tion of the proposed system. 

There still remains much work to do to refine and consol- 
idate theory and implementation of the proposed interac- 
tive and cooperative sensing and control for advanced 
teleoperation. 
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